1. Field of the Invention
This invention relates to improvements in computer software testing methods and apparatuses, or the like, and more particularly to improvements in computer software testing methods and apparatuses for interfacing suites of computer software tests to a user.
2. Relevant Background
Producers of software products generally test their products extensively to assure proper operation of their software products on as many various operating systems and computer models as are applicable and necessary. As software products become more lengthy and complex, however, the development of routines to test the software products have become of concern, especially considering the market demands of producing adequately tested software in reasonable amounts of time.
Typically, to develop a suitable test for a particular software product, a software test designer studies the product specifications, extracts a list of product features, and generates a number of "assertions" regarding the software product. An assertion is, for example, a specification of the behavior of the program, or part thereof, when it is operating properly.
For each assertion, the designer then develops a set of tests in the form of software that exercises as many as possible of the code blocks in the product to prove the validity of the assertions. Typically, a designer may generate hundreds, even thousands, of tests for a typical complex software product. The various tests are collected in sets of suites, or even subsuites, for application to the target software to be tested.
It will be appreciated that for a particular software product running tests on all possible combinations of computer models and operating systems on which the product may be run may require an enormous amount of testing time. Not uncommonly, for example, in modern software products which have thousands or hundreds of thousands of lines of code, running all of the test cases on a single operating system to prove all of the assertions throughout all of the code blocks and code paths within the product may take a number of weeks. As software products become more and more complex, it can be expected that this time will become even larger. When this time is multiplied by the total number of operating systems and computer models on which the product is to be tested, it can be seen that an enormous amount of time is required for product testing, which translates into significantly increased production cost for the product.
During the running of the various test suites required to test a particular target software application, it is difficult to tell how well the tests are progressing. For instance if a test suite has been running for 30 hours, it is difficult to tell which and how many tests have passed or failed without retrieving and examining the journal files for all of the tests. Moreover, it is not uncommon to have tests running concurrently on many machines. This makes the task of knowing the status of tests at any particular moment even more monumental.